Appendix 1: Object Classification In Speaker Verification 



Classification and Probability Density Estimation 

Speaker verification is a classification problem like any other data object vector involving two 
classes: target speakers (/) (user of object) and impostors (-/) (perpetrator of object). In order to 
do classification, in this case, a set of measurements derived from recordings of a speakers voice 
are needed. These measurements are conveniently represented as D-dimensional vectors: 

(x <= BP) 

Each speaker is characterized by a probability density function: 

which measures the likelihood of observations. The probability density is characterized by 

p(5|7) > Q.V£ < Equation: 1.1 

p(0) = J>(2|i)P(J)+?(5|-J).P(-/) < Equation: 1.2 




Equation: 1.3 



where P(l) andP(-/) are the a priori probabilities of respectively target speaker trials and impostor 

trials. For speaker verification, the a posteriori probability of the claimed speaker, I, given an 
— * 

observation, is of interest. 

The a posteriori probability can be computed by Bayes rule 



Since / and -/ are mutually exclusive we have 

P(I|J)+PH"|2) = 1 < Equation: 1.5 

i.e. the probability that the identity claim was correct given the observation, p | us the 
probability of some other speaker (not I) was speaking sum to one. It is attractive to use the a 

posteriori probability ^i^W) f 0 r classification purposes: the identity claim is accepted or 
rejected by the rule: 
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Decide ( tf WW ^ P ^ J W 

| reject otherwise 




1/ 



Figure: 1 Probability densities for the two classes, I and -I. The densities overlap in the 
regions: 



Equation: 1.6 



this causes the Bayes error rate to be greater than 0. A classifier that uses this decision rule is 
called a Bayes classifier. The error rate of a Bayes classifier is equal to 



J Li Jl^j 

= f P(-f)p(5lO^+ f P(r)p(S\I)d£ < 



Equation: 1.7 
Equation: 1.8 
Equation: 1.9 



where 
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Lt = {s i p(m > p(->i\m « 

L-,/ = {x | P(J|f) < P(-./|5)} < 



Equation: 1.10 



Equation: 1.11 



In practice the probability functions: 



P{I\x ) and P(-J|£) 



are unknown and can only be approximated. Hence, the error rate of any practical decision 
strategy is bound to have an error rate which on average is not less than the Bayes error rate. 

A Priori Probabilities & Risk Minimizations 

The average error consists of two terms; rejections of target speakers (TA errors): 



Using a posteriori probabilities to classify samples is essentially the same as classifying 
according to maximum likelihood. The overall error rate is, however, dependent on the relative 
number of impostor and target speaker trials. If impostor trials are much more frequent than 
target speaker trials, then it pays of too classify some samples as class -/ even if class / is more 
likely, because the overall absolute error is more dependent on E-l than on El. In other words, E-l 
is minimized at the expense of El. The way to balance these error rates optimally is by fixing the a 
priori probabilities to reflect the relative number of impostor/target speaker trials (object attempts). 

Assigning prior probabilities is only one way of balancing TA and IR errors. Generally the two 
types of errors may have different consequences and it may therefore be desirable to achieve a 
balance which reflects the cost of misclassification. In this case P(l) and P(-l) are replaced by: 




Equation: 1,12 



and acceptances of impostors (IR errors): 




Equation: 1.13 



C(I) = P(T)C(->I\T) <- 
<?(-./) = P(-.J)C(/h-0 * 



Equation: 1.14 



Equation: 1.15 



where ^ > * is the cost of classifying an 
according to risk and not a posteriori probability: 




-sample as I. The classification is here 



R(I\x) = 



c{ihi)p{T) P (m < 



Equation: 1.16 



Analogously to equation 1.6 we have the decision rule: 



108 



I re] 



Dfv ; (Jr» > SUCC«Pt if iS(iJx) > 4 Equation: 1.17 



reject otherwise 

A more pragmatic approach to the problem of balancing TA and IR errors is to decide a priori an 
acceptable error rate for either E/ or E. t \ and then use this to determine the decision surfaces 
(and by extension P(l) and P(-l)). Whatever way is chosen, the real problem of estimating the 
class likelihoods, 

and 

remains the same. 



Probability Estimation 

One approach to implementing a decision rule is to separately estimate the probability densities 
and 

and 

p(x\^I) 

in the test situation - use Bayes rule to convert likelihoods to probabilities, which can be used in 
place of 

P(I\x) 

This solution, however, is more extensive than required, since the verification (which by virtue of 
it's utterance translation becomes a binary data object) problem only depends on the likelihood 
ratio: 



(LR(£)): 

P(I\X) > P(-rI\£) 

t 



LR(x) = fflX &X > 1 



In terms of LR(~x), the decision function 2.6 becomes: 
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^ . , f aCCeDt ]f LH(X) > 1 < Equation: 1.18 

Decide < . . . 

I reject otherwise 



The Bayes decision surface between class / and class -/ is characterised by: 

LR(z) = 1.0 

For classification purposes we only need to know on which side of the decision surface the test 
— « 

sample * falls. In the example given in figure 2.1, this surface is the simplest possible: a single 
point x = f, where t is the' decision threshold. 

A distinction is made between parametric and non-parametric classification. The difference lies in 
the prior assumptions that are made about the class distributions. Parametric classification 
assumes that the samples to be classified belong to a narrowly defined family of probability 
density functions, whereas non-parametric classification makes only weak assumptions about the 
prior distributions. Hence, non-parametric classification is more general, whereas parametric 
classifiers are easier to construct, because they have fewer degrees of freedom. 

Parametric Classification 

As an example of parametric classification, we might assume that the classes 0* — 1> ^) are 
characterised by normal probability densities: 

In this case: 

Equation: 1.19 

LR(x) 

is given by: 

ln(LR(f)) = g(x) < b^uo 

^ * y Equation: 1.21 

This is a quadratic function. If we furthermore assume that the two distributions share the same 
covariance matrix S1 = S2 = S, this simplifies to 
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g(x) = a(x - ft) 



Equation: 1.22 




Left: The classes have similar means: 
Hi = 15.^2 = 17, 
Right: The classes have different means: 

= 15./i 2 = 27 

In the right example, the Bayes decision surface can be approximated well by a linear function, 
where 

5 = $~ l (p>i- fe) + 



This is a linear function. In discriminate analysis equation 1.22 is known as Fisher's linear 
discriminate function. As we have seen, this discriminate function is optimal for normally 
distributed classes characterized by the same covariance matrices, but its usefulness goes 
beyond this. It is a robust function, which (although not optimal) can be used with good results if 
the class distributions have the form of "spherical clouds". In fact, even if it is known that equation 
1.21 - and not equation 1.22 - is the optimal discriminate function, equation 1.22 may yield better 
results (Raudys and Pikelis 1980). The problem when 

using equation 1.21 is that from a limited sample set, it is difficult to obtain good estimates for S1 
and S2. This is especially true in high dimensional spaces. 



Equation: 1.23 
Equation: 1.24 
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The linear classifier is less sensitive to estimation errors since the dependence is primarily on the 
first order moments (the means): 



which are easier to estimate than S1 and S2 (the second order moments). If needed, the linear 
classifier may be further simplified by assuming S to be diagonal, or even S equal to the identity 
matrix. 

Example 

Figure 2 shows two examples of 1 -dimensional density functions for two normally distributed 
classes. In both examples the Bayes decision surfaces are quadratic, because the variances are 
different 



(<r| = 16, = 1) 



In case one the means are: 



and in case two: 



Hi = 15, jm 2 = 27 



Assuming equal priors, we can determine a decision rule using equation 1.21: 



LR(aO = 1 



Equation: 1.25 




Equation: 1.26 



Hence we have the decision rule: 




Class 1 ifz<15.3 V x > 18.9 
Class 2 otherwise 



The error rate is 
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E = ±(Ei + E 2 ) 
= i (0.30 + 0.07) 



w 18.8% 



In the linear case we have from 1 .22: 



LR(s) = 1 ««- 



1 



which leads to the decision rule 



Decide 



Class 1 if x < 16.0 
Class 2 otherwise 



Equation: 1.27 



Equation: 1.28 



With the error rate (f)'4® + 0.16)/2 ~ 28% The Q uac j r atic classifier is here significantly 
better than the linear classifier. In case 2 the corresponding decision rule becomes 



Decide 



Class 1 if x < 24.2 V x > 31.4 



Class 2 otherwise 

for the quadratic classifier and 



Decide 



Class 1 if x < 21.0 
Class 2 otherwise 



for the linear classifier. The average error rates are respectively 0.007% and 0.03%, which very 
small for both decision rules. Relatively, the quadratic decision rule is, however, still significantly 
more accurate. This is not because it is quadratic: a linear decision rule such as 



Decide 



Class 1 if ^ < 24.2 
Class 2 otherwise 



has the same small error rate as the quadratic decision rule. Hence, the difference in 
performance is here caused by the assumptions about the prior distributions. 
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Linear versus Non-Linear Decision Surfaces 
Assuming a priori that the solution to 

LR{X)=1 * Equation: 1.29 

is linear in % simplifies the design of a classifier. Non-linear classifiers are more powerful, 
because they allow the solution to 1.29 to be drawn from a larger set (which usually includes the 
linear solution as a special case). There is, however, nothing limiting about assuming linear 

—V — P 

decision surfaces, since the linearity refers to ^ ( but the vector 2 may be "preprocessed" before 
being given to the classifier. Assume, for instance, that the optimal decision surface - in a given 
2D problem 

(f = (x l ,x 2 ) T ) 

has the form 

Ax\ + Bx% + Cxi?2 + £>zi + Ex 2 + F= 1 

A linear classifier is able to implement this decision surface if the classification, rather than in 
terms of and x 2 is done in terms of 

<hm, mop 

where 





= 4 




= 4 


US) 


= X\X2 


US) 


= Xi 


US) 


= a? 2 



Equation: 1.30 



In other words, the 2D quadratic decision function can be implemented by a linear function in a 
5D space. 



Non-parametric Classification 

Figure 3 shows a realistic example of what the class (speaker or the object) distributions in a 
speaker recognition system or an object recognition engine might look like. • 

The assumption that the observations from a given speaker are drawn from a normal distribution 
is here reasonable. 
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Fisher's discriminate function is suitable for discrimination between any two speakers (and in this 
case comparative to object containing any given data source), but is obviously a poor model (in 
2D) for discriminating between one target speaker and the remaining speakers in the population 
(a line can not be drawn which separates an individual speaker from most of the other speakers 
in the population). In fact, the impostor class is too complicated to be modeled well by any simple 
parametric distribution. This is a common situation for many pattern classification problems. A 
number of techniques exist for non-parametric classification and probability density estimation. 



Figur 3 Probability distribution of 2D samples drawn from a set of ten different 



Non-parametric Probability Density Estimation 

Given a training set of samples with known class membership, non-parametric probability density 
estimation is the problem of constructing a PDF, that approximates the real PDF characterizing 
the classes without assuming anything about this function other than it exists. 



Histogram Rules 

The simplest approach to non-parametric density estimation is to divide the feature space into 
volumes v of size h D , where h is the side length of a D-dimensional hypercube. The likelihood of a 

given test sample, & , can then be computed by identifying the volume, v(3 ), to which it belongs, 
and computing the relative number of training samples that fall in this volume: 
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Equation: 1.31 



where n \ v \&)) \s the number of samples that fall in the volume, v \^) t to which # belongs, 
and N the total number of samples in the training set. 1.2.2 k-Nearest Neighbour. 

Nearest neighbour PDF estimation removes the problem of selecting the parameter h by letting 
the sizes of the different volumes vary so that a fixed number of training samples (k) fall in each 
volume. The result is a so called Voroni partition (tessellation) of the feature space. An example 
(k= 1) is given in figure 4 



Like the histogram rule, however, the probability density estimate is discrete: 

two neighbouring samples on different sides of a cell boundary generally have different 
likelihoods, despite the fact that the distance between them may be 




Figur 4: Voroni partition of the feature space resulting from a 1 -nearest 
neighbour 

Rule arbitrarily small. The Voroni partition also has a boundary problem, because some cells may 
have an infinite volume, which means that samples falling in these cells have an estimated 
likelihood of zero. 



Kernel Functions 
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vizi 

An alternative generalisation of the histogram rule is to compute as a sum of kernel 
functions (Hand 1982): 



Equation: 1.32 

7,1?" V h ) 

1=1 

The shape of the kernel determines the characteristics of r * ' . For instance a uniform 

kernel 



f 1 if^G[-l;lp «- 
^ ' \ 0 otherwise 



Equation: 1.33 



essentially leads to the histogram rule, whereas if ^^is a continuous function then -^^is 
continuous as well. Gaussian kernels'are a popular choice: 



Since approximates a PDF, it is convenient to require 



Equation: 1.34 



f K\x)dx = 1 



Equation: 1.35 



K{x) > 0,V£ * Equation: 1.36 

because this automatically means that is a PDF. 
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x2 



XI 



Figure 5: Likely Hood 



Likely hood 

Figure 5: Kernel estimate of the density function corresponding to figure 3 The kernel functions 
are generally placed non-uniformly in the feature space. Hence, as opposed to the simple 
histogram rule, some regions of the feature space are not "modelled" at all, and in others - where 
the density function is complicated - several kernel functions may overlap in order to model the 
density. 

For instance, to approximate the density function shown in figure 3, it would be reasonable to use 
10 kernels, with the centers corresponding to the center of each of the circular regions into which 
samples of a specific speaker fall. In this case h should reasonably correspond to the standard 
deviation of a given speakers data. An example of this is shown in figure 1 .5, where Gaussian 
kernels have been used. 



Non-parametric Classification 

The purpose of estimating PDF's is to be able to compute a postheory probabilities, which can be 
used in decision rule 1.6. It is possible, however, to implement 1.6 directly, without this 
intermediate step. The way to do this is, basically, to partition the feature space into regions and 
label each region according to which class samples falling in this region (probably) belong to. 
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It is not hard to see how the k-Nearest Neighbour rule can be used for classification: simply label 
each Voroni cell according to which class the majority of the k samples in the cell belong. The 
resulting decision surfaces will be piece wise linear. 




Figur 6: The perceptron (right) forms a hyper plane and classifies samples according to 
which side of the hyper plane they fall. 

Classifiers can also be based on kernel functions. In this case the requirements to 

the kernel functions K() are less restrictive, because the constraints of a PDF do not have to be 

fulfilled. The Radial Basis Function (RBF) network is an example of a classifier based on kernel 

functions. 

Basis Function Radius Maximisation 

For RBF networks a structure can be imposed on the basis functions by considering the radii of 
the basis functions: 



W-r (|^|) 



Equation: 1.59 



the smaller h is the more "spiked", is the basis function. A spiked basis function is only sensitive 
to a very small region of feature space and may well signify over training. Wide basis functions (h 
large) cover a large volume of the feature space; the larger h is the more the basis function 
resembles a simple bias which is always active. Hence, a network trained to have large radii is 
more likely to be able to generalise; the radii should be expanded to the point where it does not 
significantly impair the classification performance on the training set. 
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Classifier Ensembles 



It is a problem for many models - in particular neural networks - with even just a limited 
complexity, that the training algorithms used for estimating their parameters are unable to 
determine the global minimum of the optimization criteria, but only succeeds in determining a 
local minimum. For this reason it can be useful to train several classifiers on the same data, and 
use these networks to create a new "super" classifier. The combination of different networks can 
not easily be done in the parameter domain, but networks representing different local minima are 
likely to model different parts of the problem, and a classifier defined as the average output of the 
individual classifiers will in general perform better than any of the individual classifiers: if the 
individual mean square error rates (equation 1.40) of N classifiers is denoted, 



t can be shown that the expected mean square error rate of the ensemble of classifiers is given 
by (Perrone and Cooper 1994): 



provided the networks make errors independently. Hence, as long as the errors are uncorrelated, 
the performance of the classifier ensemble can be improved by adding more networks: the mean 
square error rate is cut in half each time the number of networks is doubled. 

For perceptron type models, networks representing different local minima can be created simply 
by initializing the weights differently (Hansen and Salamon 1990; Battiti and Colla 1994). In 
Benediktsson et al. (1997) individual networks (perceptrons) are trained on data that has been 
transformed using different data transforms. Ji and Ma (1997) propose an algorithm specifically 
for selecting and combining weak classifiers (perceptrons). 

Speaker Verification 

Speaker verification and object handling in a randomized environment is a pattern recognition 
problem, and conceptually it is a very simple, since only two classes (patterns) need to be 
discriminated: target speakers or object and impostors. However, it is not easy to separate the 
two classes in the feature space. The class distributions are complex and must in practice be 
modelled using non-parametric techniques. Neural networks are attractive classifiers for problems 
of this kind: their discriminative training schemes enable them to focus the modelling on the 
regions of feature space that discriminate speakers or objects well. 

A problem with many training or object learning algorithms, however, is that they are unable to 
guarantee optimal values of the model parameters. In this case structural risk minimization 
techniques can be used for placing constraints on the models that enhance their ability to 
generalize. A different approach to the problem with -sub-optimal- parameters is to use ensemble 
techniques: An ensemble of simple sub-optimal classifiers can be combined to form a new more 
powerful and robust classifier. Ensemble methods are attractive, because the error rate of the 
classifier ensemble, in principle, is inversely proportional to the number of ensemble members. 



El 



- ■ ■ ■ • 



En 




Equation: 1.60 
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Appendix 2: Obj ct Analysis Exemplified By RBF Bas d Phon me 
Mod ling 



This example presents a classifier architecture, which can be applied for speaker 
verification at the event level, however it is to be viewed as example of a method that could be 
used for any given object data type. The classifier -a RBF network - is itself not ableto identify the 
events on which it operates and relies on the feature extraction process to do this. Figure 1.1 
shows the classifier architecture schematically.Hidden Markov Models are used for segmenting 
the speech signal. A hidden Markov phoneme model, models the phoneme segments as a 
mixture of normaldistributions, where the means and covariances of the mixtures change at 
discrete points in time: at the state transitions. The discrete changes should ideally be 
continuous, but this is difficult to model. 

After the phoneme segments have been identified, a new feature extraction is performed (section 
1.1), whereby each individual phoneme segment is re-presented by a single vector of features. A 
feature vector representing an entirephoneme observation will here be referred to as a phoneme 
vector: 



When the phoneme vectors have been extracted, the signal no longer contains time information; 
the fact that the phoneme vectors were measured sequentially over a period of time is irrelevant 
and contains no information about the speaker 

identity. Further 1 the binary form of the voice print is "creafecT on a (true) random utterance 
model, which makes the binary object entirely unique. What this essentially means is that the 

vector model becomes a random vectorn 

The basic feature representation used here is in terms of filter bank energies 
and the phoneme vectors therefore need to be normalised in order 
to eliminate the signal gain (section 1.2). Following this they are subjected to 
a transformation 1 : 



Frame Selection 

Phoneme durations are a function of phoneme context, overall speech tempo 

and other factors; phoneme durations are highly variable. For a static modelling 

approach it is necessary to represent the phonemes by a fixed number of features. 

This can be done by using the Markov segmentation, where each phoneme is 

segmented into a number of sub-segments corresponding to the different emitting Markov states 

in the phoneme model. Possible representation schemes are: 





before finally being passed as input to the RBFnetwork, which computes the 
speaker probability: 
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1. Compute a new "variable" frame segmentation (and speech parameterisation), where the 
new frame length is adjusted to be an integer fraction of the total phoneme segment. 
Computationally this may be relatively expensive, but the advantage is that the entire 
phoneme segment is used. 

2. Select a fixed number (A/) of the existing frames as representatives of the phoneme 
segment. Several frame selection strategies may be considered: 

a. Linear selection: select N linearly spaced frames from the phoneme segment. 

b. Sub-segment selection: select one frame from each sub-honeme segment. In 
order to promote homogeneity of representation, the selection should be done 
consistently; e.g. by always selecting the center frames in each sub-phoneme 
segment modelled by separate HMM states. This is motivated by the hypothesis 
that center frames represent the same point in the "moving average" transition 
which the speech signal undergoes in the phoneme segment. 

c. Maximum Likelihood Selection: select the frame from each sub-phoneme 
segment that has the highest likelihood. 



After the relevant frames have been identified, the corresponding feature vectors are 
"concatenated" to form one long vector. 

Selection schemes 2A and 2B are quite similar; it has here been chosen to use 2B as the frame 
selection strategy, because in connection with ensemble methods (see section 2.7) variations in 
the frame selection strategy can be used for generating "different" phoneme models for the same 
phoneme. Selection scheme 2B can easily be varied by selecting, e.g. the right or left most 
frames in each sub segment instead of the center frame. 



Normalisation 

A problem with the filter bank representation of the speech signal is that the signal gain is not well 
controlled. The signal gain depends on the speakers speaking level, the distance to the 
microphone, the angle between the mouth and the microphone and the recording equipment. 
This effectively means that the absolute gain cannot be used for speaker recognition, and must 
be normalised. As is usual for speech processing, a logarithmic filter bank representation is used 
here. This means that the logarithm of the energy output from each filter 
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Figur 7: RBF network 

bank is used. Energy outputs below one are discarded; they most likely repre- 
sent noise and due to the singular behaviour 2 of the log function, it is best not 
to model these energies. 
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In the logarithmic energy domain, the gain factor becomes an additive bias: 

Equation: 1 



log(^ = log(5)+logf + 



Taking the log() of a vector here means that the log() function is applied to ev- 
ery vector element. Likewise, addition (multiplication) of a scalar and a vector 
means that the scaler is added (multiplied) to every vector element. Since scale is not relevant, 
phoneme vectors are assumed to have norm 1: 



XI = 



> 1=1 



Equation: 2 



after scaling the norm is 



\\sz\\ = s 



2>? = * 

*=1 



Equation: 3 



The gain can therefore be removed by computing the norm of, 
and subtracting the logarithmic norm from the filter banks put out: 



y = log{Sx) - log \\Sx\\ = logs 



Equation: 4 



To further homogenise the data, the vector: 

y 



is here normalised to have norm 1 



If an independent gain factor is associated with each filter bank channel, this results in a bias 
vector being added to the feature vectors. This type of gain can 

not be eliminated by looking at one particular feature vector, but can instead be compensated for 
by estimating the average energy output over one utterance. 

Bias removal is a useful heuristic in practise, but is actually a non-trivial problem because the bias 
which is estimated depends on the phonetic content of the utterance (Zhao 1994). This heuristic 
is not used here. 



RBF Training: 

The normalised phoneme vectors are subjected to a transformation before being 
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input to a phoneme, 



and speaker dependent RBF network, which is used for 
computing the function: 



$) = tanh 



Equation: 5 



where S is the activation function scale and 




Equation: 6 



were D is the dimensionality of the input vectors. The basis function scales, d, 
and the variances, 

<? 

, are constrained by: 



Equation: 7 



which ensures that the network will approximate the optimal Bayes Discriminant 
function: 



A number of techniques can be used for this (Press et al. 1995; Bishop 1995). In 

this case, the simplest approach is to use gradient descent, because the gradient 

here is easy to compute; because of the size of the network the training algorithm converges so 

fast that conjugate gradient, or Quasi-Newton methods are not required. Gradient descent is an 

iterative technique, where the parameters in iteration / are updated according to: 
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Equation: 8 



Equation: 9 



Equation: 10 



Equation: 1 1 



Equation: 12 



where 
OO 



OO 



E^CO = oo A^77 2 (£) < oo <■ 



and 



dwi 

dE 

9&k 



atanJi(T(^)) 



Equation: 13 



Equation: 14 



^ /w&A , ^ ataiJl(Y(^)) Equation: 14 



dE 



7 dTCWp) \ <>ik ^ Equation: 15 
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dE 

w th p=l 

dE 



^ vv^ . ^(to ^ E£,uation:18 



Equation; 16 



Equation: 17 



and 



atanh(Y(^)) = 4 

&*0) (exp(T(^)) + exp(-T(^))) : 

T(0) = 



The gradients are here shown to be computed as the summation over all the 

training samples. In order to speed the training process up, this requirement is usually relaxed so 

that subsets or even individual samples are used as the basis 

for computing the gradient and updating the parameters. This is reasonable if the training data is 
"periodic" \ 

The form of the gradient equations are relatively easy to understand. The gradient equations 
have some common terms and some specific terms. 

Common Terms All gradients include the error term, 

(M)-^) 

which is zero unless samples are misclassified. Hence, the parameters are not updated if 
samples are classified correctly. In case of misclassifications, the error term is positive if the 
target output is negative and negative if the target output is positive. The error term can be given 
a class dependent weight in order to emphasise one class error rate over the other. For instance, 
target speaker patterns may be given a higher weight, because the training set contains relatively 



1 The period should here be at least two so that a target speaker pattern and an impostor speaker pattern is 
presented in each period. More generally the -period- could be increased so that each update is based on a 
set of distinct phoneme observations -for instance corresponding to different phoneme contexts. If this this is 
not done the learning can tend to be "erratic": the network becomes biased to the most recently presented 
training token and -forgets- some of the information it has previously been taught. 
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few target speaker patterns, and hence the classifier is more likely to "over learn" these patterns 
than it is the abundant impostor speaker patterns. 



A second term which is present in all gradients is, 



This term has the effect of preventing parameter changes if, 



\?0 P )\ » 0 



i.e. if the parameters, 



p 



is misclassified by a large margin. Intuitively this is useful if the training set contains outliers, 
which can not be correctly classified by a small change of the existing parameters. 

A third term shared by all gradients is the basis function output, 



which is a value between zero and one. Hence, the parameters related to a given basis function 
are not updated unless the sample, 



is activated. 
Weights 

Weights are updated so that for misclassified samples, the weight is increased 
if the target output is positive and decreased otherwise. In the final classifier, 
basis functions with a positive weight represent class / and basis functions with 
a negative weight represent class YJl- 

Means 

Basis functions representing the target class, 





falls in the hyper elliptical region where, 



(sigp(u*i)=sign(fp)) 
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are moved closer to the misclassified sample and basis functions representing the opposite 
classare moved away. The step size depends on how "activated" the individual basis functions, 

* t 

are, the radius of the basis functions, 

the distance to the misclassified point and as usual the size of the classification error. 

Basis Function Scales 

The width of the basis functions are controlled by 

Q 

For basis functions representing the target class, 

Q 

is decreased (the width is increased) so as to include that sample in the sphere of influence of 
those basis functions. For basis functions representing the opposite class, 

Oi 

is increased (the width is decreased) so as to exclude the sample from the sphere of influence of 
these basis functions. 

Updating the variances has the same effect of widening the width of the basis functions for the 
basis functions representing the target class and decreasing the width of the basis functions 
representing the opposite class. 

Variances 
The variances, 




specify the relative variance of the individual-feature elements. The variances do not necessarily 
correspond to the statistical variances of the individual elements, but rather to the importance of 
the features. Feature components that have little importance for the classification, may be given a 
large "variance" so that they have relatively less influence on the activation of the basis function. 

Activation Function Scale 

The scale of the activation function S is increased for samples on the correct side of the hyper 
plane implemented by the perceptron, and decreased for samples on the incorrect side. The 
classification of samples, however, is not improved or changed by updating S. Consequently the 
learning algorithm does not change the value of S for the purpose of minimising the error rate. 
The activation function scale may, however, be adjusted subsequently in order to improve the 
RBF model as a probability estimator. 
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Initialisation 

The iterative training algorithm requires initial estimates of the network parameters. The 
parameters of a RBF network are much easier to interpret than the weights of a MLP, and 
consequently it is not necessary to initialise using random values. Specifically, a clustering 
algorithm can be used for computing reasonable basis functions representing respectively the 
target speaker and the cohort speakers. The weights corresponding to target speaker basis 
functions can be initialised to 

m N w 

Up** = * < Equation: 19 



where 

is the number of training samples falling in the fih target speaker cluster. Likewise the weights 
corresponding to cohort speaker basis functions can be initialised to: 



nX" 1 - 1 ) — « < Equation: 20 



The bias weight, 



should be initialised to a value less than zero: if the network is presented with a phoneme vector 
that does not activate any basis functions, the classification should be ""^ (rejection). 

The convergence of the training algorithm depends critically on the initialisation 
of the basis functions, but is in practise insensitive to the weight initialisation. 
Hence, the weights may simply be initialised to random values (in the range 

M; ID- 



Posterior Probabilities 

The RBF networks are trained to minimise the mean square error rate on the 
training set (equation 1.9). Minimisation of this error criteria causes the RBF 
network to approximate the optimal (Bayes) discriminant function given by: 

9Bwes$) = P(I$) - Phl$) * E<,uation:21 

This important fact has been proved by several authors (Ruck et al. 1990; 
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Richard and Lippmann 1991; Gish 1990a; Ney 1991). 



Even though WO approximates the optimal discriminant function, it still 
remains to answer whether or not it, in principle, is capable of exactly imple- 
menting this function. The squashing function, tanh(), present in the output, 
of the RBF network limits the number of mappings from R° to [-1;1] that can 
be implemented. For instance, a general function such as 



can not be implemented by an RBF network of the above type, even if it had 

an infinite number of basis functions. It would be unfortunate if P^9^.) was of this type, 

because that would mean that it could not, even in principle, be 

computed. 

The underlying function is, however, very flexible. By 

application of the Stone-Weierstrass Theorem it can in fact be shown that this 
function can approximate any mapping from R° to R 1 arbitrarily well (Hornik 
1989; Cotter 1990). Since tanh(x) is a monotone function which can take on 



any value in the interval [0;1], it is up to 



to approximate the function: 



T(^) = arctanh(p(I|^) - P(I\?)) 



Equation: 22 



The choice of tanh(x) as activation function is, however, not arbitrary. Con- 
sider, for instance, that in a 2-class classification problem, the two classes to be 
discriminated are characterised by Gaussian probability distributions: 



(2tt)-p/2|U-,/|V2 



Equation: 23 



Equation: 24 



According to Bayes rule, the a posteriori probability of class / is given by: 
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where 



P {<j/\i)p(i) +p(^h/)j>(-j) 
1 



1 + ££^M£jbn 



l-hexp(-2a) Equation: 25 

| tajih(o.) -+- | 



+ 0.5(^-w) T U7*(<?-w)) 

- 0.5(^ - j3-./) T U^(^ - •* Equation: 26 

This is exactly the form we would like it to have, since if the RBF network 
approximates the discriminant function: 

g$) = P(I$) - P(^I\$) +— Equation:27 

then we have (using 2.5): 
where 



1 /r„ 1 

= 2^(0 ) + 2 * Equation: 28 

= -\90)+\ * Equation: 29 
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g{4?) = tanh(T(^)) * Equation: 30 

Adjusting the Activation Function Scale 

As probability estimates, equations 33 and 34, are somewhat crude. If a 

steep activation function (large activation function scale S) is used, the output is 

essentially a binary variable. The activation function scale (S) may be adjusted 

by first estimating the empirical activation function from - ideally - an independent test set: 

Q0) = ^E^)-^))-[i-^Ee(T(^)-r(4))] 



1 JV> 1 N_j 



Equation: 31 



where is a step function 



1 ifs>0 
0 otherwise 



Equation: 32 



and where ' " " " ' < n.,->/J " • ■ iYN-,j,-*I are the phoneme vectors 

in the independent test set. Now the value, s * 1 ' , for which > / * is identified 

(i.e. G(? Pl ) = 2P(I) - 1)) 

and the activation function scale adjusted so that 
- Equation: 33 

This is done by choosing: 

— Equation: 34 

* = 

where 
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ai'CtSlILll(a?) = * Equation: 35 



An alternative, and potentially more accurate approach is to simply replace tanh() by the empirical 
activation function (equation 36). 

An alternative, and potentially more accurate approach is to simply replace 
tanh() by the empirical activation function (equation 36). 



Adjusting the Bias 

Training a RBF network from a limited training set is difficult. The problem is 

usually not the impostor part of the training set, but rather the target speaker 

part. This, of course, can in itself make it difficult to train a speaker model, but in particular it 

makes it difficult to adjust the model so that it achieves the desired balance between the TA and 

IR errors. The balance can to some extent be controlled by various training parameters, eg. by 

scaling the error term 

r . differently for target speaker samples and cohort speaker samples, by 
presenting target/cohort patterns with different frequencies, or by the way the models are 
constrained using weight/radii penalties. These means are, however, fairly crude, and a more 

(wq) 

accurate approach is to adjusting the bias ' . of the 

RBF models. This can be done by estimating the mean and variance of 

given the target speaker, ^(^1-0 arK j g j ven the impostor speakers, ^^l" 1 ^). 
Assuming a Gaussian distribution of these two variables, the bias is reduced 

(knew = btfd — Ab) 

' , so that, 

— Equation: 36 



= B 



This solution can be found by determining the roots of: ^ 

J * Equation: 37 

d.iy + &.^ui!l.i|. 21ll (f>). w=0 

where the following shorthand was used: 

Equation: 38 
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x = T$) 

m = T(<^|i) * Equation: 39 

/i2 = Var(T(<^|J)) < Equation: 40 

For B = 1 this is the same equation as equation 1.26, (the example on object classification. The 
solution we are interested in is the one between ^~(«/m-0 and T(4*\~ '-0 

An alternative - if the Gaussian assumption is poor - is to use the empirical activation function 
(equation x.36). If a different balance, S, of errors is desired, the bias can be adjusted according 
to: 

<HT«0) + 1 „< Equation: 41 

-G(T0)) + 1 " 



t 



^ Equation: 42 

G(¥(<j})) = — " 1 < Equation: 43 

B 4- 1 



Hence, to adjust the odds ratio to have balance B, the solution, \r / <( t0 equation 

48 is determined and subtracted from the bias: 



1, ^ r * \ to ' 



For B = 1 the equal error rate is approximated, for B < 1 the number of TA 
errors is minimised at the expense of the IR errors, and for B > 1 the IR errors 
are minimised at the expense of the TA errors. 



Figure 8 shows an example where the class conditional empirical distribution functions, 
and 

jwfl-j)) 

and the empirical activation function, 
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for a set of speaker models. The figure shows the functions both 



ErprcaJ D rtntirtn a Fwtura £agri£nfo£w) 




Figur 8 Empirical distribution functions 



5 




For the training data, respectively 1622 and 6488 local target speaker and impostor speaker 
decisions were used. For the test data, respectively 394 and 1576 local decisions were used. 




Figur 9 Empirical distribution functions after bias compensation 



for the training data and for the test data. For thejraining data the empirical 

activation function is approximately zero for , but not for the test data (the speaker 

models are "overtrained"). Figure 9 shows the same functions as Figure 8, but after bias 
compensation. 

In summary, a phoneme based speaker model has been described. The model uses HMM's as 

"feature extractors" that represent phoneme observations as fixed vectors (phoneme vectors) of 

spectral feature elements; this part of the model is speaker 

independent. The phoneme vectors are transformed and finally passed as input 

to a phoneme dependent RBF network, trained to estimate the speaker probability from the 

phoneme vectors. The speaker probability can be used directly 

for producing a (local) speaker verification decision, or it can be combined with 
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other speaker probabilities estimated from other phoneme observations in order 
to produce a more robust decision. The input vector (phoneme) is only stated to exemplify what 
an object based i.e. verification could be. Any other type of biometric vectors could be used with 
training filters accordingly. 
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Appendix 3: Obj ct Bas d D cision Making Ex mplified By Speaker 
Verification 



Object verification - or in this case speaker verification is a binary decision problem, and can 
therefore in the end be reduced to computing a score and verifying identity claims by determining 
whether or not the score is greater or less than a given threshold, f: 



When computing this score or i.e. an object value, each phoneme segment in the speech signal 
makes a contribution (even when phonemes are not explicitly modelled). In a conventional text 
independent speaker verification algorithm, the contribution of the different phonemes to the 
overall score (e.g. utterance likelihood) is unknown; the overall score depends on the particular 
frequency with which the phonemes are represented in the test utterance, and on the duration of 
each phoneme segment. 

This is clearly not optimal, since no regard is taken to the extent that local scores contributed by 
individual phoneme segments express speaker identity and the extent to which different 
phonemes express the same information about the speaker; e.g. a nasal and a vowel presumably 
represent information which is largely complimentary whereas two back vowels, say, represent 
highly correlated information about the speaker. 

The algorithm described here has two parts: first phoneme segments are identified 
and the speaker identity modelled for each phoneme segment independently. The result of this is 
a number of local scores -one for each different phoneme in an utterance - which subsequently 
must be combined in order to produce a global verification decision or a class of object data. 



Combining Scores 

An RBF networks are trained to approximate the discriminant function: 



Decide 




if score > t 
otherwise 



equation: 1 




equation: 2 



where 




is a phoneme observation. Since: 
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P{I\h + P(rl\fo = l 



equation: 3 

we have 




which can be used for implementing a decision rule for a single phoneme observation. When 
several independent phoneme observations are available, more robust decisions can be made by 
combining the local scores into a global score. Two basically different approaches can be 
followed: ensemble combination and probability combination. 



Ensemble Combination 

One approach to combining local verification scores is simply to "average" the local scores: 



score = Ji<|> ^ ^ &Mj) 

2^=1 W^i 2=1 j=l 

equation: 6 
where 

is the number of different phonemes in the alphabet, 

the number of observations of phoneme 1 and the-/ observation (phoneme 
vector) 

of phoneme fr . It is a characteristic of this scoring rule that for an increasing 
number of observations, the score will converge to a value in the range [-1; 1]; 
The magnitude is not directly affected by the number of observations. 
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Probability Combination 

An alternative to ensemble combination is to exploit the fact that the networks compute a 
posteriorj_probabilities^ When several independent observations, 

$(0 = <£, . . . <L 

' L are made, the confidence of the classification is expected to rise. 

This can be expressed by defining the odds ratio: 



equation: 7 
since 

P(7|«< r 5) + P(-J|*W) = l 

equation: 8 
it follows that 




P(-J|*W) 



1 

equation: 9 & equation: 10 

Hence, an alternative scoring strategy is to use 

score = 

equation: 1 1 

It is a characteristic of this scoring rule that in practise it will converge to either -1 or +1 when 
more phoneme observations are added. 

The difference between equation 6 and 1 1 is mainly the assumption about 
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the independence of the observations. Suppose for a given phoneme vector, 

the speaker probability is estimated to, say, i*<t\&) — o.r if equation 1 1 

(probability combination) is used we assume that the probability is only 0.7 and not 1.0, because 

the observation has been affected by "random" noise, whereas if equation 1.6 (ensemble 
combination) is used, we assume that a certain proportion of the impostor population is capable 

of producing phoneme vectors like 

This distinction is important, because noise can be "averaged" (derived) away, whereas obtaining 
more observations (of the same event) cannot be expected to improve the probability estimate, if 
the same impostor speakers are fundamentally able to produce the same phoneme vectors as 
the target speaker. 

A problem with both equation 1.6 and 1.11 is, however, that the overall score will be dominated 
by the most frequently occurring phoneme. This is unreasonable to the extent that different 
phonemes can be regarded as different sources of speaker information (Olsen 1997b; Olsen 
1996b). 

In practise it is, however, possible to use equation 1.6 and 1.11 with good results, because 
"pathological" sentences that are dominated by a specific class of phonemes are not frequently 
occurring. Any reasonable sentence will typically have a broad selection of phonemes 
represented, but it should still not be left to chance how to weight the evidence provided by each 
phoneme observation. 



Committee Machines 

Each phoneme model can be regarded as a speaker verification expert given a specific type of 
information: observations of a specific phoneme. Since individual experts are assumed to model 
different "aspects" of the speaker, it makes sense to limit the influence each expert can have on 
the global score. One approach to this is to use either equation 1.6 or 1.1 1 for combining the local 
scores from the same expert into a phoneme level local score. A local binary decision - with an 
empirically known probability of being correct - can then be made for each phoneme represented 
in the test utterance: 

1 if P(J|*0 > 0-5 
—1 otherwise 

equation: 12 




Following this approach, the simplest way of combining local decisions into a 
global decision, is by making a "majority" vote: 

score = ^ 
equation: 13 
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Probability of Correct Majority Decision 




2 4 6 8 10 12 

No Classifiers 

Figure: 1 The probability of a committee machine making a correct decision as a function of the 
number of committee members. 



where is the number of different phonemes represented in the test utterance. This type of 
global classifier is called a committee machine (Nilsson 1965; Mazurov et al. 1987). 

If the individual decisions are independent and all have the same probability, P, of making a 
correct decision, the probability of the committee machine making a correct decision is given by: 



A=|JV/2I4-1 \ / 



k=lN/2j+l 

equation: 14 



where N is the number of committee members. The probability function * comm 

(AT) 

IS 

shown in figure 1. The graph is "rippled" because for even /V, a tie = j S counted 

as an error even though the error probability is actually only 50%. As long as the errors are 
uncorrelated, the performance of the committee machine can be improved by adding more 
members. Provided P > 0:5, the committee machine always performs better than the individual 
committee members. 

This is not necessarily the case if the individual classifiers have different classification accuracies, 
but the model is nevertheless remarkably robust in this case. Assume, for instance, that three 
classifiers with individual accuracies P1; P2 and P3 are to be combined. The committee machine 
performs at least as well as the most accurate of the individual classifiers (say PI), provided: 
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Pi < P1P2P3 + PlW - P3) + Pld ~ P2)PZ + (1 - Pl)i>2P3 



Pl< 



equation: 15 & equation: 16 




For instance if P2 = P3 = 0.9, then P1 must have an accuracy higher than 0.99 
if it alone is supposed to be more accurate than the combination of P1, P2 and 
P3. 



Expert Weighting 

Votes from different experts are not equally important; the different phoneme 
dependent speaker models have different accuracies. The basic voting scheme 
can therefore be improved by weighting the individual votes differently. A "static" 
approach to this would be to simply weight each vote by the expected equal 
accuracy rate, A E er = 1 - EER, of the corresponding classifier: 



(*0 = { _; 



D ( ^ ] = j A BER if P(/|*0 > 0.5 
L ^ zi 1 otherwise 



equation: 17 

The corresponding "dynamic" weighting scheme would be to weight each vote 
by the differential speaker probability computed by the classifier: 

equation: 18 

Even if the probability estimate ^(^l^)js somewhat crude, the advantage here 
is that the weight is dependent on the actual phoneme observations. 

Expert Grouping 

Phonemes can be divided into different groups, e.g. nasals, fricatives, plosives, 
vowels etc. Two experts specialising on, say, two nasal phonemes are intuitively 
likely to show correlations in the voting domain, whereas two experts specialising 
of different phonemes, say, respectively a nasal and a fricative phoneme, are less 
likely to show correlations. It may therefore be reasonable to divide the experts 
into groups representing different phoneme classes. A speaker verification score, 
Dc;l, can then be computed for each phoneme groupfCJ; 
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£>c,l($,) 



f 1 if z£[P(I|$ t )>*7 



— 1 otherwise 



V 



equation: 19 



where #C denotes the number of phonemes in group C. Equation 19 effectively defines a new set 
of experts. The global verification decision can then be made as before by combining the votes 
from the group experts, rather than from the "phoneme" experts. In principle this decision strategy 
can be extended, to include several layers of experts, where the experts at the lowest level 
represent different individual phonemes and experts at the upper levels represent broader sound 
classes (nasals, vowels, fricatives, etc.). 

Modelling Expert Votes 

An attractive way of combining N expert votes is to train a network (RBF or MLP) to learn the 
empirically best combination strategy (Wolpert 1992). This way both the accuracy of the individual 
experts and the correlation between different expert votes can be taken into account directly. 
When this approach is followed, all that has taken place up to the point where the expert votes 
must be combined is essentially regarded as feature extraction; the feature vectors are here 
decision vectors: 



equation: 20 

There are, however, two problems with this approach. 

• The first problem is that the "super" network, which combines local expert votes, can not 
be trained on decision vectors produced simply by evaluating the local experts on the 
data on which they were trained -the experts are likely to be over trained and their - 
training data votes- are therefore too "optimistic". Hence, either additional training data 
must be provided or alternatively the super network must be speaker independent. 

• The second problem is that here the local expert votes represent different phonemes and 
the phonetic make of different test utterances can vary a lot, and this makes it impossible 
to train a network that optimally combines the votes resulting from particular test 
utterances. 



Given a limited number of training utterances, it is of course possible to simulate a much larger 
number of decision vectors by combining relevant expert decisions extracted from different 
training utterances. However, the number of possible phoneme combinations that can occur is 
still very large. Suppose, for instance, that in any given utterance, exactly 15 different phonemes 
out of 30 possible will be represented. Then up to 



3um = (^ ri) ) ? ... 5 ^(^ ) )) 



T 
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different vote combinations would have to be considered. This calculation ignores that votes may 
be based on more than one phoneme observation - and hence be more reliable - and that the 
actual number of different phonemes may be more or may be less than 15. 

A possible solution to this dilemma is to make the super classifier utterance specific, i.e. to 
postpone the training until the moment it is decided which prompting text to issue next - or even 
more convenient: until a phoneme segmentation has been computed for the actual speech 
utterance. The super classifier may in this case be a simple perceptron, and the training is 
therefore not in itself a serious Computational problem. Figure 2 shows an example of this. 

Alternatively - in order to avoid the iterative perceptron training algorithm - Fisher's linear 
discriminant function can be used for learning the individual expert weights. 

In summary, this example discusses how local speaker probabilities estimated from 
individual phoneme observations (which essentially is an object can be combined in order to 
produce global speaker verification decisions. Successful combination schemes must take into 
account that on the one hand some specific phonemes are more informative than others, and on 
the other hand that different phonemes to some extent provide complimentary information about 
a speaker. 

The main difficulty faced when deciding how to weight each local decision is that -unless the 
prompting texts given to speakers are seriously constrained - the total number of different 
phoneme combinations that can occur in test utterances is extremely large. Hence, these weights 
can not easily be computed a priori. 
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*S ri, =# jl- 



•F°=* K~ 





Figure: 2 A super classifier 

The classifier takes the differential speaker probabilities from the individual phoneme models as 
input and combines them into a global score: 
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